[Data] Add fillna and dropna functions #54844
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
There was a problem hiding this comment.
Pull Request Overview
This PR adds two essential null handling functions (fillna and dropna) to Ray Data, bringing it to feature parity with pandas and PySpark for basic ETL operations and ML dataset preprocessing.
Key changes include:
- Implementation of
fillnamethod to replace missing values with scalar or column-specific values - Implementation of
dropnamethod to remove rows containing missing values with flexible filtering options - Comprehensive test suites covering edge cases and different data types
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
python/ray/data/dataset.py |
Adds public API methods fillna and dropna with comprehensive documentation and examples |
python/ray/data/_internal/logical/operators/map_operator.py |
Implements logical operators FillNa and DropNa for the execution framework |
python/ray/data/_internal/planner/plan_udf_map_op.py |
Implements planning functions with PyArrow-based transformations for null handling |
python/ray/data/_internal/planner/planner.py |
Registers the new logical operators with their planning functions |
python/ray/data/tests/test_fillna.py |
Comprehensive test suite for fillna functionality |
python/ray/data/tests/test_dropna.py |
Comprehensive test suite for dropna functionality |
python/ray/data/BUILD |
Adds build targets for the new test files |
Comments suppressed due to low confidence (6)
python/ray/data/tests/test_fillna.py:105
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
python/ray/data/tests/test_dropna.py:53
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
python/ray/data/tests/test_fillna.py:192
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
python/ray/data/tests/test_dropna.py:104
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
python/ray/data/tests/test_fillna.py:220
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
python/ray/data/tests/test_dropna.py:208
- [nitpick] The variable name 'i' is not used in the loop body. Consider using '_' instead to indicate it's unused.
for i, (actual, exp) in enumerate(zip(rows, expected)):
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
bot
added
the
stale
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
|
Adding comment to remove label. Also to mention larger changes to the code structure to add these as operators. |
Signed-off-by: soffer-anyscale <173827098+soffer-anyscale@users.noreply.github.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
github-actions
bot
added
unstale
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
- Add advanced filling methods: forward, backward, interpolate - Add limit parameter for consecutive fill operations - Add ignore_values parameter for custom missing value definitions - Add comprehensive parameter validation and error handling - Update documentation to match Ray Data style (remove framework references) - Add comprehensive test coverage with 35 test cases covering: * All new parameters and methods * Edge cases and error conditions * Multi-block behavior * Schema preservation * Performance testing Files modified: - dataset.py: Enhanced API with new parameters - fillna_operator.py: Advanced logical operator with validation - dropna_operator.py: Enhanced with ignore_values support - plan_fillna_op.py: Physical implementation of advanced methods - plan_dropna_op.py: Support for custom missing values - test_fillna.py: Comprehensive test suite (18 tests) - test_dropna.py: Comprehensive test suite (17 tests) Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
- Format code according to Ray's black 22.10.0 standards - Fix formatting in logical operators and planners - Update test file formatting Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
- Add Any to typing imports to resolve NameError when using List[Any] type annotations - Fixes test failures caused by missing type import Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
- Fix SystemException usage in exceptions.py (remove incorrect exception chaining) - Add missing Optional import in plan_fillna_op.py - All files now pass lint checks and follow proper import conventions Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
- Notebooks were reformatted by the pre-push hook linter - No functional changes, just formatting updates Signed-off-by: soffer-anyscale <stephen.offer@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Null handling is a basic and common ETL requirement that other data frameworks have. It is important for feature parity and for ML dataset preprocessing to have common null handling features.
This PR adds
ds.fillnaandds.dropnafunctions, modeled after Pandas and PySpark functionalities.Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.